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ABSTRACT 


The eight science and engineering practices outlined in the Next Generation Science Standards 
(NGSS) are key to student science learning. However, there is limited guidance for teachers on 
what these practices should “look like” in instruction across different grade ranges and topic 
areas. 


This paper describes the development of two products to facilitate measurement of the extent to 
which classroom instruction aligns with the Framework for K—12 Science Education (National 
Research Council, 2012) and the subsequently released Next Generation Science Standards 
(Next Generation Science Standards [NGSS] Lead States, 2013), particularly the science 
practices. The first product is a primer that operationalizes the science practices for different 
grade levels and content areas. The second product is a selected-response survey for measuring 
how often teachers implement the practices in their science instruction. 
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INTRODUCTION 


In 2013, the National Research Council (NRC) released the report Monitoring Progress Toward 
Successful K-12 Education: A Nation Advancing? The report calls for a national indicator 
system that could be used to improve science, technology, engineering, and mathematics 
(STEM) education in the United States, and sets forth 14 indicators related to students’ access to 
quality learning, educators’ capacity, and policy and funding initiatives. However, as the report 
states, a great deal of work needs to be done, both in terms of developing measures for a number 
of the indicators and establishing a system for collecting the data. 


There is a great need for methods of measuring Indicator 5, namely the extent to which 
classroom instruction aligns with the Framework for K—12 Science Education (National 
Research Council, 2011) and the subsequently released Next Generation Science Standards 
(Next Generation Science Standards [NGSS] Lead States, 2013). The NGSS describe three 
dimensions—Disciplinary Core Ideas, Crosscutting Concepts, and Science and Engineering 
Practices—that form the foundation of student opportunity to learn science and engineering. The 
three dimensions are intertwined such that students learn targeted content ideas through engaging 
in the practices to develop explanations for real-world phenomena. The eight practices outlined 
in the NGSS—asking questions/defining problems; developing and using models; planning and 
carrying out investigations; analyzing and interpreting data; using mathematics and 
computational thinking; constructing explanations/designing solutions; engaging in argument 
from evidence; and obtaining, evaluating, and communicating information—are, thus, critical to 
students understanding science and engineering. 


Unfortunately, there is little existing guidance for teachers on how students should experience 
the NGSS practices in K-12 instruction. Further, there are no tools that can be used cost 
effectively on a large scale for measuring the extent to which classroom instruction provides 
opportunities for students to engage with the practices. This paper describes the development of 
two products to facilitate measurement of Indicator 5 in the NRC report. The first product is a 
primer that describes key elements of the science practices' and provides illustrative examples of 
these key elements of instruction across different grade bands (K—2, 3—5, 6—8, 9-12) and subject 
areas (e.g., biological sciences, Earth sciences, physical sciences). The second product is a 
selected-response survey for measuring how often teachers implement the practices in their 
science instruction. Together these resources will (a) provide much needed guidance on how the 


' Presently, project efforts are focused only on the science practices. However, it is possible that this work may 
expand to the engineering practices in the future. 
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practices can be implemented in K-12 classrooms, and (b) afford a means of monitoring 
progress towards widespread student engagement with the science practices. 


PROCEDURE 


Two methods of data collection were used to inform development of the primer. First, an 
extensive review of existing literature was conducted on how the practices can be incorporated 
into classroom instruction. Second, an online, modified-Delphi panel (Miller & Pasley, 2012) 
was convened to identify key aspects of instruction that students need to experience for each of 
the science practices at different grade bands and areas of science. Information gained from the 
literature review and panel process were used to write the primer. 


This information was also used to guide the development of survey items assessing how often 
teachers implement the science practices in their instruction. The item-development process 
includes cognitive interviews with teachers, expert review of the items, and a large-scale pilot to 
determine the psychometric properties of the instrument. These activities are described further in 
the following sections. 


Literature Review on the Practices 


A literature review was conducted to identify and summarize current research and practice-based 
knowledge focused on engaging students with the science and engineering practices. A search 
utilizing the ERIC and Google Scholar databases was conducted to find articles on the NGSS 
practices published since the release of the Framework for K—12 Science Education (NRC, 2011) 
as we assumed that relevant work that preceded it was already incorporated in the Framework. 


Key words and phrases (e.g., science and engineering practices, NGSS, models, argumentation, 
explanations) were used in conjunction, and the initial search resulted in 76 empirical and 
practitioner articles and conference papers. These resources were reviewed more carefully and 
resources that included only general descriptions of the practices were removed from the 
literature pool. The remaining 47 articles were coded by the practices, grade levels, and subjects 
addressed, when possible. 


The majority of the articles that passed the screening were from practitioner journals (35 
articles). Most of the practitioner articles were intended for high school teachers (16), followed 
by middle school (12), and elementary (7). Frequently these practitioner articles provided an 
illustration of how a practice might play out in a classroom, but with little guidance about the key 
features of the illustration. The remaining 12 articles were either empirical or were products of 
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conference presentations. The descriptions of the practices culled from these articles were used 
as a foundation for the questions posed in the subsequent panel. 


Expert Panel 


An expert panel was convened to “unpack” the science practices, identifying key elements of the 
practices in different grade bands and areas of science. The expert panel was composed of 
individuals with backgrounds and experiences in different grade levels and content areas. In 
addition, panelists had varied levels of involvement with the NGSS, including some panelists 
who served on the writing team. 


The panel process involved four rounds of questions that were emailed to the panelists. 
Questions in Rounds One and Two addressed four practices: constructing explanations; engaging 
in argument from evidence; obtaining, evaluating, and communicating information; and using 
mathematics and computational thinking. Rounds Three and Four focused on the remaining 
practices: developing and using models, asking questions, planning and carrying out 
investigations, and analyzing and interpreting data. Panelists were asked to complete a series of 
questions/tasks related to each practice including: (1) constructing a definition of the practice; (2) 
identifying the key elements of what students need to experience at various grade levels and in 
different science content areas that are essential for them to understand the practice; and (3) 
generating examples of what instruction “looks like” as students engage with the practice. 


Data gathered during the expert panel process were analyzed in an ongoing fashion during the 
data collection period. The first round of panel responses were analyzed in order to frame the 
second round of questions, an iterative process that continued through all rounds of the panel. 
Use of the three strategies (constructing a definition of the practice, identifying key elements of 
the practice, and generating examples of students learning science through engaging with the 
science practices) led to a purposeful accumulation of data and vetting of emerging themes. The 
goal in data analysis was to identify and verify key elements of a practice, meaning that they 
were ideas that panelists largely agreed were important for the practice under discussion. 


In each round of the panel, responses to each question posed produced a data set that was 
analyzed. Project staff constructed a simple thematic coding scheme for analyzing each item in 
each round and tracking the strength of agreement (e.g., similar responses for an item). In 
addition to the reflections that panelists offered in each round of the panel, the examples that 
they provided were treated as data to be coded. These examples were analyzed as short 
narratives of particular practices in terms of the relationships illustrated or explicated. 
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Developing Teacher Survey Items 


An initial set of 42 survey items was drafted based on what was learned through the literature 
review and expert panel process. Nine items asked how much emphasis teachers place on 
practice-related learning objectives for their students. For example: students gaining proficiency 
in developing scientific models (physical, graphical, or mathematical representations of real- 
world phenomena) to help develop explanations, identify questions, make predictions, or 
communicate to others. Thirty-three items focused on the frequency with which teachers have 
students engage in specific practice-related activities, such as determining what data would need 
to be collected in order to answer a scientific question. 


Teachers in grades K-12 were then recruited to participate in a first round of cognitive 
interviews (Desimone & Le Floch, 2004; Hamilton, Nussbaum, & Snow, 1997) around these 
items. The cognitive interview process involved a teacher reading and thinking aloud as s/he 
answered the survey items. The interviewer then probed regarding particular item features (e.g., 
the clarity of the wording, how terms were interpreted). For example, teachers were presented 
with an item that asked how frequently students revise their explanations for real-world 
phenomena based on additional evidence, and then prompted by the interviewer to describe how 
they interpreted the term “explanations” in this context. The point of the cognitive interviews 
was to maximize item validity, as teachers’ responses revealed whether they understood the 
items. 


Data from the cognitive interviews were used to edit the item pool. Researchers either (a) 
accepted an item as written, (b) made edits to address the issues identified in the interviews, or 
(c) drafted a completely new item. New and substantially revised items then went through a 
second round of cognitive interviews. 


As of the writing of this paper, the second round of cognitive interviews had just concluded. The 
next step in survey development consists of two pilot studies. The first will be a large scale, 
online administration of the items targeting 3,000 teachers across the grade ranges. A variety of 
Statistical analyses will be conducted with the pilot data. First, descriptive statistics will be used 
to identify items with low discrimination (1.e., most teachers selecting the same response option). 
Lack of variation is an indication that an item may not be performing as intended; any such items 
will be reviewed and possibly dropped from the item set. Second, the sample will be randomly 
split into two groups to conduct factor analyses. Exploratory factor analysis will be conducted 
on one group to identify potential scales and items that can be dropped (e.g., because they do not 
associate strongly with any one scale). Confirmatory factor analysis will be conducted on the 
other group to determine the quality and appropriateness of the final factor solution. Both the 
exploratory and confirmatory factor analyses will examine whether the factor solution varies by 
grade band. 
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The second pilot will involve approximately 40 teachers who will allow researchers to observe a 
week of their science instruction. At the end of the week, the teachers will complete the survey 
and then participate in an interview about their use of the practices during that week. Data from 
the surveys, interviews, and researchers’ field notes will be compared to assess the validity of the 
survey items. 


FINDINGS 


The process of developing the primer surfaced a number of important points about both the 
discipline of science and school science. Additionally, a number of lessons were learned about 
teachers’ understanding of the science practices through survey development. Findings in both 
of these areas are described in the following sections. 


Primer 


The literature review and panel process gave rise to several critical principles that guided the 
development of the primer. These principles became the foundation for how the science 
practices are ultimately defined, described, and illustrated in the primer. 


> The ultimate goal of science is the development of evidence-based explanations for, and 
models of, the natural world. The science practices are in the service of these goals, and 
critical to students gaining an understanding of how scientific knowledge is generated. 


Although the practices are intimately intertwined, the unpacking process highlighted the 
danger that the ultimate goal of science could get lost by focusing on individual practices. 
We think it is important that teachers keep this ultimate goal of science in mind when 
planning and delivering instruction so that students develop an understanding of how the 
practices are in service of this goal. The science practices are critical to achieving this 
goal, as students need to understand that scientific evidence is generated through a 
systematic and social process. 


>» The science practices apply to all fields of science and grade levels. 


Throughout the rounds of the expert panel, panelists consistently agreed that (a) the 
practices do not differ across topics/subject areas and that (b) student engagement in the 
practices grows progressively more complex as grade level increases. However, as the 
key elements of the practices were further unpacked, it became apparent that some key 
elements are too sophisticated for students in the lowest grade bands to engage with in 
authentic ways, particularly some aspects of mathematics and computational thinking. It 
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also became evident that as students progress through the grade bands, it may become 
unnecessary for teachers to explicitly engage student with some of the key elements of 
practices since students will have sufficiently engaged with particular aspects of a 
practice in earlier grades. For example, assuming students have learned to distinguish 
scientific and non-scientific questions in previous years, high school teachers will likely 
not have to emphasize this aspect of asking questions. 


> There is variation in how teachers, teacher educators, researchers, and other stakeholders 
view the nature and role of the practices. 


The panel process revealed substantial differences in how people define the science 
practices and describe “what it looks like” for students to engage with a particular 
practice. Additionally, it quickly became apparent that key terms are used in different 
ways. For example, panelist varied in their interpretations of what constitutes “claims,” 
“arguments,” and “explanations.” For this reason, the primer includes a glossary of key 
terms used. It is hoped that creating consensus around the use of key terms will foster 
productive conversations around the practices. 


» The practices often have overlapping elements and are used concurrently in science. 


Because the practices overlap as they are used in science, it was sometimes difficult for 
panelists to tease apart key elements for individual practices. As such, the work of 
developing the definitions and key elements described in the primer was necessarily 
iterative in nature. However, it is certainly inauthentic to consider key elements and 
practices in isolation, as the practices are often used in connection with each other. For 
this reason, the primer includes vignettes that illustrate how instruction that incorporates 
the key elements might appear in different subject areas and grade bands. 
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> Students need to have opportunities to reflect on their use of the practices. 


In addition to having opportunities to engage in the science practices, it became apparent 
that students should also have periodic opportunities to engage in metacognition about 
their learning and be asked to explicitly reflect on their use of various practices. For 
example, if students use different types of models (e.g., descriptive, relational) for 
various purposes (e.g., to make predictions, to compare with other models) over the 
course of their 8" grade science class, the teacher may want to help students reflect on the 
various types models they have used and discuss the purpose of models more broadly in 
science. 


Teacher Survey Items 


Although piloting of survey items is still underway, the cognitive interview process revealed a 
number of insights about teachers’ understanding and interpretation of the science practices. The 
information gleaned from the interviews provided a lens for making edits to the items, towards 
the ultimate goal of creating a valid and reliable instrument for measuring teacher 
implementation of the practices in their science instruction. 


> There was variation in how teachers defined and interpreted terms that were directly 
related to student engagement with the science practices. 


Consistent with lessons learned in the process of creating the primer, cognitive interviews 
revealed that teachers defined terms used in the survey items in many different ways. For 
example, teachers were not sure about the extent to which their students had opportunities 
to gain proficiency in argumentation because they did understand what argumentation 
entails. Some teachers interpreted argumentation as a means of defending an 
explanation, while others interpreted it as a negative way of communicating (e.g., being 
“argumentative”). To address this issue, it was necessary to add definitions to some 
items in order to clarify the intended meanings of widely misinterpreted terms. 


> Many teachers were unclear about what constitutes a model. 


Although interviewed teachers were generally familiar with physical models (e.g., clay 
structures representing the parts of an atom, mobiles of the solar system) they were less 
certain about other types of models that might be used in science classrooms, such as 
drawings, graphs, or simulations. This lack of familiarity with various types of models 
and how they are used in science was particularly true of teachers at the elementary level. 
Therefore, survey items were reworded to draw attention to various types of models and 
the purposes they can serve. 
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>» Teachers were unsure whether engaging in the science practices requires students to 
design, carry out, and revise each step of an investigation on their own. 


When thinking about the ways in which their students engaged with the science practices, 
teachers interviewed were unsure whether it was necessary for students to design, carry 
out, and revise each step of an investigation on their own. For example, teachers 
wondered whether it “counted” if students revised investigational procedures developed 
by someone else rather than revising procedures that they initially developed on their 
own. Similarly, teachers wondered if it “counted” for students to conduct investigations 
that they did not design themselves. To address this issue, information was added to the 
items to clarify the range of ways in which students might engage in key elements of 
planning and carrying out investigations. The goal of the revised item wording was to 
make it clear to teachers that students can authentically experience this practice in a range 
of ways. 


>» Teachers conflated mathematics and computational thinking, often because they did not 
know what computational thinking is. 


Teachers struggled the most with survey items focused on mathematics and 
computational thinking. In many instances, this was because teachers did not know what 
counted as “computational thinking.” For example, teachers at the elementary level often 
interpreted computational thinking to mean doing computation: addition, subtraction, 
multiplication, and division. Secondary teachers tended to think that students were 
engaged in computational thinking any time they used a computer. Other teachers 
indicated that they had no idea what the term meant. Because of this lack of familiarity 
with computational thinking, teachers admitted that they read past the term computational 
thinking when it was presented in the survey items, and answered strictly based on their 
knowledge of mathematics. To address this issue, the items were revised so that teachers 
were asked to consider either mathematics or computational thinking in a single item, but 
not both. Further, information was added to the items to clarify what computational 
thinking means. However, the second round of cognitive interviews found that these 
edits were not sufficient to overcome the confusion around this practice; consequently, 
these items were removed from the survey. 


DISCUSSION 


NRC’s Monitoring Progress report highlights a number of areas in which development work is 
needed to realize the goal of an indicator system for K-12 STEM education. The products of 
this project will address one of those areas of need—measuring the extent to which classroom 
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instruction provides students opportunities to engage with the practices described in the NGSS 
(Indicator 5). 


The primer created to guide survey development will provide a source of much needed guidance 
on how the practices can be implemented in classrooms. Further the primer contains information 
that could potentially be used to inform the development of instructional materials, professional 
development programs, and assessment tasks that integrate the science practices in grade- and 
content-appropriate ways. 


The survey items being developed will facilitate the indicator being measured on a large scale in 
a cost-effective manner, which is critical for monitoring progress and identifying needs for 
teacher professional development. 
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